Expecting and Achieving Gains in Student Performance: 

EVAAS Value-Added Report 2002-2004 for the Archdiocese of Indianapolis 

By Ronald W. Costello, Ed.D., and Barbara Shuey 

In the present environment of accountability, (No Child Left Behind - NCLB), schools 
are being held accountable to improve student performance based upon academic standards. This 
creates both opportunities and challenges for our Catholic schools. An opportunity will be given 
to parents from failing public schools to choose another school for their child. Voucher programs 
are already in place Washington, D.C., and others, like Indiana, are considering legislation on 
this topic. The challenge will be to prove that our Catholic schools are successfully improving 
student performance. We believe the following model will help guide our efforts. 

Through a school improvement grant from Lilly Endowment Inc., the Archdiocese of 
Indianapolis has been able to partner with the Milken Lamily Loundation to implement the 
Teacher Advancement Program (TAP) in eight of our schools. TAP connects the elements of 
professional staff development, improved student growth and performance-based compensation. 
The archdiocese has been using the Terra Nova (CTB McGraw-Hill) standardized test to 
measure student gains in language, math and reading to measure the success of TAP and other 
initiatives. We have tested approximately 5,000 students in 22 schools in grades 2 nd through 9 th 
over the last three years. The test results have been used to determine if student performance is 
changing at the student, teacher, school, and district level. Here are the findings and explanation 
of the value-added model we have been using. 
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Method 



Over the three years, we have compared test results using the value-added model developed by 
Bill Sanders through SAS EVAAS Institute in North Carolina. The assumption of this model is 
that for a student to show growth, that student must achieve a gain score that is greater than one 
year’s growth. The performance model was developed by Sanders and used in the state of 
Tennessee to measure student performance. The Sanders value-added model is a mathematical 
(statistical) comparison of the student test data over time to determine gains in performance. As 
stated in the fact sheet from the SAS Institute, value-added gives us “more than a snapshot of 
current performance levels. It provides a precise measurement of student progress over time and 
a reliable diagnosis of opportunities for growth.” (Fact Sheet, 2003) 

Findings 

After three years, the archdiocese is ready to report the performance findings based upon 
the 22 schools involved in the Lilly Endowment initiative using the value-added model. This is 
done by computing a district gain and standard error from the base year 2001-2002 in each of the 
three subject areas tested — language, math, and reading. Then standard score is computed from 
three years of performance data. Shown in the table below are the value-added scores for 
language (4), math (5), and reading (3). The value-added rubric score was developed by the 
Milken Family Foundation to be used to report the Sanders value-added standard score. The 
description of the values is described below. 
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Archdiocese of Indianapolis Terra Nova Value-Added District Report 



Year 


Subject 


Reference 

Gain 


District 

Gain 


Std. 

Error 


Std. 

Score 


Value- 

Added 

Score 


2001-02 


Language 


100.0 










2002-03 


Language 


100.0 


139.9 


17.2 






2003-04 


Language 


100.0 


127.1 


14.1 


! 1.9 [ 


1 4 


2001-02 


Math 


100.0 










2002-03 


Math 


100.0 


72.9 


7.9 






2003-04 


Math 


100.0 


128.4 


8.2 


3.5 i 




2001-02 


Reading 


100.0 










2002-03 


Reading 


100.0 


101.1 


14.7 






2003-04 


Reading 


100.0 


113.7 


15.3 


0.9 


3 



Milken Rubric for Value-Added Scores 

Score of 1 = The district gain is 2 standard errors below the reference gain. 

Score of 2 = The district gain is 1 standard errors below the reference gain. 

Score of 3 = The district gain is neither 1 standard error above or below the reference gain. 
Score of 4 = The district gain is 1 standard errors above the reference gain. 

Score of 5 = The district gain is 2 standard errors above the reference gain. 



The results from the value-added report are very encouraging because significant gains 
were achieved in both language and math. In the value-added model, student performance gain in 
language and math is greater than the expected one year’s growth. In math, it is twice what 
would be expected. For reading, we missed having the reading score being a value-added score 
by 0.1. 



In order to achieve significant gains in all subject areas, it is important to use the data to 
focus our improvement efforts. Using EVAAS district report, the first step would be to break 
down the testing results. Using the grade level report, allows us to identify grade levels and/or 
subject areas where performance could be improved. In our specific case the identified focus 
area is reading. 



Focusing on reading scores by grade level, the standard score is converted to one of four 
value-added scores using the criteria below. In the next chart, district-level student results are 
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shown for grades 3, 4, 5, 6, 7, and 8. At grades 4, 5, and 7, student performance is in the green 
zone. For two grades (3 and 6), the student gain score is in the yellow zone. For grade 8, the 
student performance is in the red zone. 



Using the data to direct our interventions, the reading data would lead us to conclude that 
8 th grade students achieved less than the expected one year’s growth. Students at grades 3 and 6 
had a negative gain but not as critical as the 8 th grade. A guiding question should be: If we 
improved student performance on the Terra Nova at the 8 th grade level, would the improvement 
at this grade level be enough to move the district reading score to a gain score high enough to 
show improvement in reading for the archdiocese? The answer would be yes. 
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2004 Reading by Grade Level 



2.0 



1.0 



0.0 



- 1.0 



0.9 



Green 

Zone 



3 




4 


5 




6 


-0.5 


Yellow Zone 


Grade Levi 


4 



- 0.8 




Red Zone 



- 1.2 



-2.0 J 



G Green Zone Estimated mean gain equal to or greater than reference gain. 

Y Yellow Zone Estimated mean gain below reference gain by less than one standard error. 

R Red Zone Estimated mean gain below reference gain by at least one, but less than two, standard errors. 
R* Ultra-Red Zone Estimated mean gain below reference gain by at least two standard errors. 
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Finally, we need to remember that using district-level performance data only gives a 
general overall assessment. To achieve the student performance gains we desire it is critical to 
analyze the data at the school, teacher and student level. At these levels, we can detennine which 
areas are most important to be addressed for each student. It must be kept in mind the overall 
goal is to have perfonnance gains for all students. 

Implications 

We are presenting student performance at the district level to help the readers understand 
how the EVASS value-added model works. With this working knowledge, you will be able to 
understand the meaning of the value-added report received from SAS EVAAS at the other 
levels — school, teacher and student. Schools will be able to use this information to develop or 
improve plans with interventions and strategies to address student needs. 

Following are some of our concerns and recommendations in using the value-added 
model in the future. 

> It has been difficult for the archdiocese to follow a pure research model in implementing 
initiatives from the Lilly Endowment grant program. As we have found interventions that 
work, we have made those opportunities available to all schools. This has made it 
impossible to say that we have control schools where we can compare performance data 
to initiative schools. It has never been our intent to not to improve the perfonnance of all 
students in order to do pure research. 
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> Indiana uses the Indiana Statewide Test of Education Progress-Plus (ISTEP+) to measure 
student performance on the Indiana academic standards. In the past, only students in 
grades 3, 6, 8 and 10 were tested. We used the Terra Nova test to have enough 
performance data for each grade level to do the value-added analysis. This school year 
(2004-2005) the ISTEP+ test was administered to all students in grades 3 through 10. We 
will be able to use our Terra Nova test results from the 22 schools and move to an 
ISTEP+ value-added report. The EVAAS model allows you to use the three year 
performance history without starting over. In this transition, we will begin to include the 
ISTEP+ data from the other 5 1 schools in the archdiocese. Once we are able to create 
reports based upon three years of ISTEP+ testing, the archdiocese will be able to create 
value-added reports for all 72 schools with approximately 24,000 students. 

> The Indiana fall testing forces us to link student performance back to previous school 
year. For example, the third grade test really measures perfonnance at the end of 2 nd 
grade. In order to get the linkages with to the teachers that provide the student with the 
instruction that relates to the ISTEP+ test score, we link the test to the teacher from the 
previous school year. This is manageable as long as the teacher does not have a large 
number of students enrolling new in the fall and cannot be linked to the previous school 
year. 

> With the gubernatorial change in Indiana, there has been a great deal of discussion about 
moving the ISTEP+ from a fall to spring test. The value-added model would adjust for 
this change. Our concern is that if this happens, there needs to be a plan for this transition 
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to guarantee that we do not lose a year of test data. For example, the year the change is 
made, students should be tested in the fall and spring. 

Conclusion 

The value-added model has allowed us to change how we compare student performance. 
In the past, we were comparing one student’s achievement with another, not whether the student 
was improving. When you think about this, it tells us very little about whether the school has 
made any difference in improving individual student performance. The value-added model is a 
long-term commitment to assessment. Should we not expect all students to improve more than 
what is expected in a single year? If we do, then we can say that they have added value to their 
performance! We believe that the value-added model allows us to measure and detennine if our 
Catholic schools are improving student performance. 
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